Dynamic modification of audio playback in games

ABSTRACT

A method for dynamically modifying audio playback of a video game is provided. The method includes ascertaining a game music data associated with the video game. The game music data may include a plurality of game soundtracks classified according to predetermined criteria. The method may include collecting feedback over a period of time while the user is playing the video game. The method may continue with determining, based on the feedback, one or more replacement soundtracks based on criteria associated with the one or more replacement soundtracks. The method may further include dynamically modifying the game music data while the user is playing the video game by replacing at least one of the plurality of game soundtracks with the one or more replacement soundtracks to obtain modified game music data. The modified game music data may be provided to the user while the user is playing the video game.

CROSS-REFERENCE TO RELATED APPLICATIONS

This continuation application claims the priority benefit of U.S. patentapplication Ser. No. 15/710,733 filed on Sep. 20, 2017 and titled“Dynamic Modification of Audio Playback in Games,” which is herebyincorporated by reference in its entirety.

This application is also related to U.S. Pat. No. 10,661,175, issued May26, 2020, entitled “Intelligent User-based Game Soundtrack,” thedisclosure of which is incorporated herein by reference.

BACKGROUND Technical Field

This disclosure generally relates to computer games and entertainmentapplications. More particularly, this disclosure relates tointelligently modifying audio playback of a game while a user is playingthe game.

Background

Conventionally, a video or computer game played by a user is accompaniedby one or more game soundtracks to enhance the user's gaming experience.The one or more game soundtracks can be predetermined for a particulargame by game developers. The one or more game soundtracks can bepredetermined based on actions taken by the user while playing the game.For example, the one or more game soundtracks can be provided accordingto rules set by the game developers and cannot be modified by the user.However, the one or more game soundtracks may not be a good fit for aparticular user (e.g., the game soundtracks may be repetitive, notappealing to the taste of the particular user, or may even negativelyaffect the performance of the user). Furthermore, a user isconventionally unable to provide any explicit feedback as to whether thecurrently played game soundtrack is satisfactory to the user. Inaddition, conventionally, no determination as to the effect of thesoundtrack on the user is made.

SUMMARY

This section is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription section. This summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used as an aid in determining the scope of the claimedsubject matter.

In one example embodiment of this disclosure, there is provided a methodfor dynamically modifying audio playback of a video game. The method maycommence with ascertaining game music data associated with the videogame. The game music data may include a plurality of game soundtracksclassified according to predetermined criteria. The method may includecollecting feedback over a period of time while the user is playing thevideo game. The method may continue with determining, based on thefeedback, one or more replacement soundtracks based on criteriaassociated with the one or more replacement soundtracks. The method mayfurther include dynamically modifying the game music data while the useris playing the video game by replacing at least one of the plurality ofgame soundtracks with the one or more replacement soundtracks to obtainmodified game music data. The modified game music data may be providedto the user while the user is playing the video game.

In another embodiment of this disclosure, there is provided a system fordynamically modifying audio playback of a video game. The system mayinclude a processor, a collection unit, an analyzing unit, and a memorystoring a processor-executable code. The processor may be configured toascertain a game music data associated with the video game. The gamemusic data may include a plurality of game soundtracks classifiedaccording to predetermined criteria. The collection unit may beconfigured to collect feedback over a period of time while the user isplaying the video game. The analyzing unit may be configured todetermine, based on the feedback, the one or more replacementsoundtracks based on criteria associated with the one or morereplacement soundtracks. The processor may be further configured todynamically modify the game music data while the user is playing thevideo game by replacing at least one of the plurality of gamesoundtracks with one or more replacement soundtracks to obtain modifiedgame music data. The modified game music data may be provided to theuser while the user is playing the video game.

In yet another embodiment of this disclosure, there is provided anon-transitory processor-readable medium having instructions storedthereon, which, when executed by a processor, cause the processor toimplement the above-outlined method for dynamically modifying audioplayback of a video game.

Additional novel features of example embodiments are set forth in thedetailed description, which follows, and can be apparent to thoseskilled in the art upon examination of the following description and theaccompanying drawings or may be learned by production or operation ofthe examples. The objects and advantages of the concepts may be realizedand attained by means of the methodologies, instrumentalities, andcombinations particularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example and not limitation in thefigures of the accompanying drawings, in which like references indicatesimilar elements.

FIG. 1 shows an example system architecture for dynamically modifyingaudio playback of a video game, according to one embodiment.

FIG. 2 is a schematic representation of elements of a system fordynamically modifying audio playback of a video game, according to anexample embodiment.

FIG. 3 shows a process flow diagram illustrating a method fordynamically modifying audio playback of a video game, according to anexample embodiment.

FIG. 4 shows an example computer system that can be used to implement atleast some operations of a method for dynamically modifying audioplayback of a video game, according to an example embodiment.

Like reference characters indicate similar components throughout theseveral views of the drawings. Skilled artisans will appreciate thatelements in the figures are illustrated for simplicity and clarity andhave not necessarily been drawn to scale. For example, the dimensions ofsome of the elements in the figures may be exaggerated relative to otherelements to help to improve understanding of various embodiments of thedisclosure. In addition, common but well-understood elements that areuseful or common in a commercially feasible embodiment are often notdepicted in order to facilitate a less obstructed view of these variousembodiments of the disclosure.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the presented concepts. Itshould be understood, however, that the presented concepts may bepracticed without some or all of these specific details. In otherinstances, well-known process operations have not been described indetail so as to not unnecessarily obscure the described concepts. Whilesome concepts will be described in conjunction with the specificembodiments, it will be understood that these embodiments are notintended to be limiting.

The disclosure is generally directed to a technology for intelligentlyand dynamically modifying at least a portion of audio playback of avideo game. Overall, the technology is directed to enhancing gamingexperience and to addressing drawbacks known in the art. The disclosedtechnology is applicable to computer and video games, including onlinegames, network games, mobile games, multiplayer games, virtual realitygames, and the like, which are collectively referred to herein as gamesor video games. The technology can be implemented by a computing device,a gaming device, a cell phone, a server, and so forth.

When a user plays a game, such as an online game, a network game, amobile game, a multiplayer game, a virtual reality game, and the like,game music may be provided to the user to enhance gaming experience ofthe user. The game music may include at least one game soundtrackprovided by developers of the game, and may be played in the gameaccording to rules predefined by the game developers. A system fordynamically modifying audio playback of a video game of the presentdisclosure may ascertain data associated with the game soundtrack of thegame (for example, in a form of metadata of the game soundtrack). Theascertained data may be analyzed to extract audio parameters of the gamesoundtrack. The ascertained data may be classified based on audioparameters of the game soundtrack.

The audio parameters can include one or more of the following: abeats-per-minute (BPM) value, a tonality value, a pitch value, a timbrevalue, a harmonics value, a loudness value, a rhythm value, a soundenvelope, a music genre, a release date, an emotional classification,and so forth. The emotional classification (e.g., aggressive, mellow,calm, melancholic, joyful, etc.) of the game soundtrack can bedetermined based on the analysis of the audio parameters. In someimplementations, a machine learning system can be used to classify thegame soundtrack. The machine learning system can be trained based onhistorical user data such as prior music classification. As such, theemotional classification can indicate that the game soundtrack (or acertain portion of the game soundtrack) is associated with one ofpredefined emotional states, such as happiness, sadness, joy, wonder,surprise, fear, anxiety, violence, anger, sorrow, disgust, distress, andso forth.

Moreover, the system of the present disclosure may collect feedbackwhile the user is playing the game. In an example embodiment, thefeedback includes an express action of the user (e.g., pressing a “like”button by the user to show that a specific game soundtrack currentlyplayed in the game is satisfactory to the user), evaluation of gameresults (e.g., the user achieves better results in the game when a gamesoundtrack having an “aggressive” emotional classification is played),performance of the user in the game (e.g., the user is less prone toerror when one or more game soundtracks added by the user into apreferred list is played), an emotional state of the user (e.g., theuser appears more satisfied when a game soundtrack having a “mellow”emotional classification is played), and the like. The one or more gamesoundtracks may be further classified based on the feedback.

The system of the present disclosure may also access a database, such asa music library, a music online service, music stored on a computer ofthe user, and the like. The database may have a number of soundtracks.The system may classify the soundtracks stored in the database based onaudio parameters of the soundtracks. The audio parameters may alsoinclude the emotional classification of the soundtracks that can bedetermined based on the analysis of the audio parameters. In someembodiments, the analysis of the soundtracks in the database may bepreliminarily performed so that the soundtracks may already beclassified when the user starts playing the game.

The game music may be modified by replacing at least a portion of thegame soundtrack with the soundtrack stored in the database. Thereplacement may be performed based on matching the audio parameters andthe classification of the game soundtrack with the audio parameters andthe classification of the soundtrack stored in the database. Thereplacement of game soundtracks or portions of the game soundtracks maybe performed dynamically while the user is playing the game based on thefeedback continuously collected during a gameplay of the user.

The following detailed description of embodiments includes references tothe accompanying drawings, which form a part of the detaileddescription. Approaches described in this section are not prior art tothe claims and are not admitted to be prior art by inclusion in thissection. Reference throughout this specification to “one embodiment,”“an embodiment,” “some embodiments,” “some implementations” or similarlanguage means that a particular feature, structure, or characteristicdescribed in connection with an example implementation is included in atleast one embodiment of the disclosure. Thus, appearances of the phrases“in one embodiment,” “in an embodiment,” “in some embodiments,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofembodiments may be combined in any suitable manner in one or moreimplementations. In the instant description, numerous specific detailsare provided, such as examples of programming, software modules, userselections, network transactions, hardware modules, hardware circuits,hardware chips, and so forth, to provide a thorough understanding ofembodiments. One skilled in the relevant art will recognize, however,that the embodiments can be practiced without one or more of thespecific details, or with other methods, components, materials, and soforth. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringaspects of the disclosure.

Embodiments of this disclosure will now be presented with reference toaccompanying drawings which show blocks, components, circuits, steps,operations, processes, algorithms, and the like, collectively referredto as “elements” for simplicity. These elements may be implemented usingelectronic hardware, computer software, or any combination thereof.Whether such elements are implemented as hardware or software dependsupon the particular application and design constraints imposed on theoverall system. By way of example, an element, or any portion of anelement, or any combination of elements may be implemented with a“processing system” that includes one or more processors. Examples ofprocessors include microprocessors, microcontrollers, Central ProcessingUnits, digital signal processors, field programmable gate arrays,programmable logic devices, state machines, gated logic, discretehardware circuits, and other suitable hardware configured to performvarious functions described throughout this disclosure. One or moreprocessors in the processing system may execute software, firmware, ormiddleware (collectively referred to as “software”). The term “software”shall be construed broadly to mean processor-executable instructions,instruction sets, code segments, program code, programs, subprograms,software components, applications, software applications, softwarepackages, routines, subroutines, objects, executables, threads ofexecution, procedures, functions, and the like, whether referred to assoftware, firmware, middleware, microcode, hardware descriptionlanguage, or otherwise.

Accordingly, in one or more embodiments, the functions described hereinmay be implemented in hardware, software, or any combination thereof. Ifimplemented in software, the functions may be stored on or encoded asone or more instructions or code on a non-transitory computer-readablemedium. Computer-readable media includes computer storage media. Storagemedia may be any available media that can be accessed by a computer. Byway of example, and not limitation, such computer-readable media caninclude a random-access memory (RAM), a read-only memory (ROM), anelectrically erasable programmable ROM, compact disk ROM (CD-ROM) orother optical disk storage, magnetic disk storage, solid state memory,or any other data storage devices, combinations of the aforementionedtypes of computer-readable media, or any other medium that can be usedto store computer executable code in the form of instructions or datastructures that can be accessed by a computer.

For purposes of this patent document, the terms “or” and “and” shallmean “and/or” unless stated otherwise or clearly intended otherwise bythe context of their use. The term “a” shall mean “one or more” unlessstated otherwise or where the use of “one or more” is clearlyinappropriate. The terms “comprise,” “comprising,” “include,” and“including” are interchangeable and not intended to be limiting. Forexample, the term “including” shall be interpreted to mean “including,but not limited to.”

The terms “game” and “video game” can be construed to mean acomputerized game, a computer game, a network game, an online game, amultiplayer game, a virtual reality game, a game for mobile devices, andthe like. The terms “user” and “player” can be used interchangeably andmean a person who plays a game, and also uses or participates inproviding audio playback as described herein. The term “soundtrack”should be construed to mean a sound, music, melody, song, sound clip, orany combinations thereof.

Referring now to the drawings, example embodiments are described. Thedrawings are schematic illustrations of idealized example embodiments.Thus, the example embodiments discussed herein should not be construedas limited to the particular illustrations presented herein, ratherthese example embodiments can include deviations and differ from theillustrations presented herein.

FIG. 1 shows an example system architecture 100 for dynamicallymodifying audio playback of a video game, according to one exampleembodiment. System architecture 100 can include an electronic device 105such as a game console, gaming device, computing device, mobile device,cellular phone, smart phone, tablet computer, desktop computer, laptopcomputer, workstation, multimedia device, television device, smarttelevision device, and the like. As such, electronic device 105 mayinclude input and output devices (not shown) to enable a user 130 (alsoreferred to herein a player) to play games (i.e., computer games). Forexample, electronic device 105 may include at least one speaker (notshown) configured to output (play) various sounds such as gamesoundtracks and modified game music data generated by a method fordynamically modifying audio playback of a video game as describedherein.

Electronic device 105 may also include a processor 110 and a memory 115.Memory 115 can store processor-executable instructions for execution byprocessor 110. Memory 115 can also store data associated with one ormore games or gaming applications, one or more game soundtracks, one ormore user settings or preferences, and so forth. Theprocessor-executable instructions can cause processor 110 to implementat least some operations of the methods for dynamically modifying audioplayback of a video game as disclosed herein. Electronic device 105 mayfurther include a user interface (not shown), such as a graphical userinterface, enabling the user 130 to interact with the game, activate ordeactivate the method for dynamically modifying audio playback of avideo game, adjust settings associated with modification of gamesoundtracks and displaying identifiers associated with currently playedgame soundtracks, and the like.

In some embodiments, the game can be an online game, network game, ormultiplayer game. Accordingly, electronic device 105 of the user 130 canbe operatively connected to one or more servers 120 or electronicdevices 105 of other users 135 via a communications network shown as anetwork 125. The network 125 can refer to any wired, wireless, oroptical networks including, for example, the Internet, intranet, localarea network, Personal Area Network, Wide Area Network, Virtual PrivateNetwork, cellular phone networks (e.g., packet switching communicationsnetwork, circuit switching communications network), Bluetooth radio,Ethernet network, an IEEE 802.11-based radio frequency network, Internetprotocol communications network, or any other data communication networkutilizing physical layers, link layer capability, or network layer tocarry data packets, or any combinations of the above-listed datanetworks.

In some implementations, server 120 can implement some or all operationsof the method for dynamically modifying audio playback of a video gameas described herein. For example, electronic device 105 can allow theuser 130 to play a game which generates game music, while server 120 canperform some or all operations for dynamically modifying game music databy replacing at least one of a game soundtrack with the one or morereplacement soundtracks, and cause electronic device 105 to playmodified game music data while the user 130 is playing the game. Inother implementations, all operations of the method for dynamicallymodifying audio playback of a video game can be implemented byelectronic device 105. However, it is not necessary that electronicdevice 105 is incorporated into a single housing. Rather, in someimplementations, electronic device 105 can include several units. Forexample, processor 110 and memory 115 can be incorporated into a gameconsole, while speakers for presenting the replacement game soundtrackscan be incorporated into a television set, and the game console and thetelevision set can be operatively connected to one another.

FIG. 2 is a schematic representation of elements of a system 200 fordynamically modifying audio playback of a video game, according to anexample embodiment. The system 200 may comprise a processor 210, acollection unit 220, an analyzing unit 230, and optionally, a storageunit 240, also referred to herein as a memory. The processor 210 may beconfigured to ascertain a game music data associated with the videogame. The game music data may include a plurality of game soundtracksclassified according to predetermined criteria. The game music data maybe ascertained in a form of metadata of the game soundtracks. Themetadata may include at least the audio parameters of the gamesoundtracks.

The collection unit 220 may be configured to collect feedback over aperiod of time while the user is playing the video game. In an exampleembodiment, the feedback may include one or more of the following: anexpress action of the user, evaluation of game results, an emotionalstate of the user, and so forth. The emotional state of the user mayinclude one or more of the following: a biometrical response, a facialexpression, a facial motion, a stress level of the user, and so forth.The express action may include pressing a “like” button, pressing a“thumbs up” button, pressing a “thumbs down” button, enabling a pop-up(where the pop-up may show at least a title of a song provided to theuser in the modified game music data), disabling the pop-up, and soforth. In an example embodiment, the stress level of the user may beidentified based on user vital parameters or user motions. In otherwords, the stress level of the user may be attributed to a specific gamesoundtrack, the game soundtrack may be associated with a specific typeof a biomedical response of the user, an emotional state of the user maybe attributed to a specific game soundtrack, and the like. The gamesoundtrack may be classified based on the audio parameters and thefeedback.

In some example embodiments, the classification of the game soundtrackmay include the emotional classification. The emotional classificationof the game soundtrack can be determined based on the audio parameters,game music data (metadata), gameplay, game content, language associatedwith gameplay (e.g., conversations of virtual characters), in-game soundtriggers, and so forth.

The analyzing unit 230 may be configured to determine, based on thefeedback, the one or more replacement soundtracks based on criteriaassociated with the one or more replacement soundtracks. Specifically,the replacement soundtracks may be obtained from a music source, such asa music library, a music online service, a cloud-based music service, acloud-based music storage, music files stored on a computer of the user,and so forth. The music source may include a plurality of soundtracksthat may be used as replacement soundtracks for the video game. Theanalyzing unit 230 may access the music source. The music source caninclude music files or logs (identifiers, indexes) of music files thatmay be used as replacement soundtracks. The analyzing unit 230 maydetermine audio parameters for each of the one or more replacementsoundtracks. The audio parameters may include a BPM value, a tonalityvalue, a pitch value, a timbre value, a harmonics value, a loudnessvalue, a rhythm value, a sound envelope, a music genre, a release date,an emotional classification, and so forth.

The analyzing unit 230 may further classify the one or more replacementsoundtracks based on the audio parameters. The analyzing unit 230 mayfurther index the one or more replacement soundtracks based on theclassifying to provide replacement soundtrack indexes. The replacementsoundtrack indexes may be used to match a classification of the one ormore replacement soundtracks with a classification of the plurality ofgame soundtracks. The classification of each of the plurality of gamesoundtracks and the classification of the one or more replacementsoundtracks may be determined by a machine learning system. The machinelearning system may be trained based on historical user data stored inthe historical data database. In some example embodiments, theclassification of the replacement soundtracks may include the emotionalclassification based on the audio parameters of the replacementsoundtracks.

Generally speaking, once the feedback associated with the gamesoundtracks played in the game is obtained, the replacement soundtracksare analyzed so as to match the classification of the game soundtracksand the classification of the replacement soundtracks. In other words,the music source may store one or more replacement soundtracks havingthe classification that matches the classification of a certain gamesoundtrack (or a portion of the game soundtrack). For example, when theclassification of a certain game soundtrack is a song having a“sorrowful” emotional characteristic, the analyzing unit 230 may selectone or more songs (i.e., replacement soundtracks) from the music sourcethat possess same or similar characteristics.

As such, the system 200 can analyze replacement soundtracks of the musicsource similarly to the analysis of the game soundtracks. In otherwords, each replacement soundtrack can be analyzed to extract audioparameters (e.g., a BPM value, a tonality value, a pitch value, a timbrevalue, a harmonics value, a loudness value, a rhythm value, a soundenvelope, a music genre, a release date, and so forth) to determine theemotional classification of the replacement soundtracks based on theanalysis of the audio parameters. The emotional classifications can befurther used to index the replacement soundtracks.

The processor 210 may be configured to dynamically modify the game musicdata while the user is playing the video game. The modification may beperformed by replacing at least one of the plurality of game soundtracksor a portion of the game soundtrack with one or more replacementsoundtracks or a portion of the replacement soundtrack to obtainmodified game music data.

The processor 210 may be configured to provide the modified game musicdata to the user while the user is playing the video game. The modifiedgame music data provided based on the feedback collected while the useris playing the game may help to improve the performance of the user, aswell as the emotional state of the user, and may result in bettersatisfaction of the user with the gameplay.

It should be noted that the user may not always know which replacementsoundtrack is currently playing. Therefore, the user may desire to makeinquiries concerning a title or a musician name performing a particularreplacement soundtrack. For these reasons, the processor 210 may beconfigured to temporarily display, within the video game, via agraphical user interface of an electronic device, an identifier of thereplacement soundtrack used to replace the game soundtrack. Theidentifier can include, for example, a soundtrack title and a name ofmusician.

In an example embodiment, the processor 210 may be configured to storethe feedback to a historical data database. The historical data databasemay store a plurality of feedback associated with a plurality of userscollected while each of the plurality of users is playing the videogame. In an example embodiment, the historical data database may bestored in the storage unit 240.

In a further example embodiment, the processor 210 may be furtherconfigured to search in a historical data database for one or more usershaving parameters similar to parameters of the user. The processor 210may select one or more game soundtracks recommended or preferred by theone or more users. In this example embodiment, the dynamic modificationof the game music data may further include replacing at least one of theplurality of game soundtracks with the one or more game soundtracksrecommended or preferred by the one or more users.

In an example embodiment, the processor 210 may be configured toidentify sound triggers based on a gameplay of the video game, actionsof the user in the video game, or actions of further users in the videogame. The replacement of the at least one of the plurality gamesoundtracks with the one or more replacement soundtracks may beperformed at times associated with the sound triggers. In someembodiments, the replacement means that the game soundtrack iseffectively muted, and the selected replacement soundtrack is providedinstead of the muted game soundtrack.

It should be noted that the process of replacing game soundtracks withreplacement soundtracks is performed in real-time. Moreover, thereplacement can be performed selectively meaning that only certainportions of game soundtrack are replaced with replacement soundtracks.For example, the system 200 can identify predefined sound triggers basedon a gameplay of the user, actions of the user in the game, or actionsof other users in the game that can be obtained from the historical datadatabase, and replace the game soundtrack with the replacementsoundtrack at times associated with sound triggers.

The replacement of game soundtracks with replacement soundtracks can be“seamless” to the user. To this end, the system 200 can intelligentlymix one or more game soundtracks (which are not replaced or muted) withone or more replacement soundtracks (which are used to replace certaingame soundtracks) such that a transition between these soundtracks isblended. For example, the transition may involve matching BPM values ofthe game soundtrack and the replacement soundtrack, followed by fadingin and fading out of the soundtracks.

Furthermore, the processor 210 can prioritize some replacementsoundtracks in the music source based on the feedback associated withactions of the user collected in association with the replacementsoundtracks. For example, the processor 210 can prioritize replacementsoundtracks recently played in the game, or replacement soundtracksrecently added by the user to a wish list. In other embodiments, theprocessor 210 can determine user music preferences based on a frequencyof playing the replacement soundtracks and prioritize the replacementsoundtracks accordingly. As such, the replacement of a game soundtrackcan be performed with a replacement soundtrack that has a higherpriority for the user than other replacement soundtracks.

FIG. 3 is a process flow diagram showing a method 300 for dynamicallymodifying audio playback of a video game, according to an exampleembodiment. The method 300 can be performed by processing logic thatincludes hardware (e.g., decision-making logic, dedicated logic,programmable logic, application-specific integrated circuit), software(such as software run on a general-purpose computer system or adedicated machine), or a combination of both. In one example embodiment,the processing logic refers to one or more elements of electronic device105 of FIG. 1 . Operations of method 300 recited below can beimplemented in an order different than described and shown in thefigure. Moreover, method 300 may have additional operations not shownherein, but which can be evident to those skilled in the art from thedisclosure. The method 300 may also have fewer operations than shown inFIG. 3 and described below.

The method 300 may commence with ascertaining game music data associatedwith the video game at operation 305. The game music data may include aplurality of game soundtracks classified according to predeterminedcriteria. The video game can be implemented by an electronic device, aserver, or both to allow the user to play the game. For example, thevideo can include a computerized game, video game, network game,multiplayer game, or an online game. The game soundtracks can bepredefined by game developers.

At operation 310, feedback may be collected over a period of time whilethe user is playing the video game. In an example embodiment, thefeedback may include one or more of the following: an express action ofthe user, evaluation of game results, an emotional state of the user,and so forth. The express action of the user may include pressing a“like” button, pressing a “thumbs up” button, pressing a “thumbs down”button, enabling a pop-up (in which the pop-up shows at least a title ofa song provided to the user in the modified game music data), anddisabling the pop-up. The emotional state of the user may include abiometrical response, a facial expression, a facial motion, a stresslevel, and the like. The stress level of the user may be identifiedbased on user vital parameters or user motions.

The feedback may be stored to a historical data database. The historicaldata database may store a plurality of feedback associated with aplurality of users collected while each of the plurality of users isplaying the video game.

The method 300 may continue with determining, based on the feedback, oneor more replacement soundtracks based on criteria associated with theone or more replacement soundtracks at operation 315. In an exampleembodiment, the one or more replacement soundtracks may be determined asfollows. The method 300 may include accessing a music source that mayincorporate a plurality of soundtracks that can be used as replacementsoundtracks. Audio parameters for each of the one or more replacementsoundtracks may be determined. The one or more replacement soundtracksmay be classified based on the audio parameters. The one or morereplacement soundtracks may be indexed based on the classifying toprovide replacement soundtrack indexes. The replacement soundtrackindexes may be used to match a classification of the one or morereplacement soundtracks with a classification of the plurality of gamesoundtracks. The classification of each of the plurality of gamesoundtracks and the classification of the one or more replacementsoundtracks may be determined by a machine learning system trained basedon historical user data stored in the historical data database.

The game soundtracks may be classified by extracting audio parameters ofthe game soundtracks and classifying the game soundtracks based on theaudio parameters. The classification of the game soundtracks can befurther based on the following feedback collected during the gameplay ofthe user: a current emotional state of the user, a current stress levelof the user, a current emotional state of the gameplay, and so forth.The current emotional state of the user can be determined by capturingimages of a user facial expression with a video camera; determining,based on the images, whether the user is blinking and how tense the userlooks; and the like. The feedback may be analyzed using the machinelearning system. Alternatively, the current emotional state of the usercan be determined by recognizing a speech of the user or by analyzingin-game user actions. The current stress level of the user can beidentified based on user vital parameters or user motions. For example,keystroke dynamics, user body temperature, user heart pulse, userin-game actions, and other parameters can be analyzed to detect thecurrent stress level of the user. The vital parameters of the user maybe measured using sensors of the electronic device of the user.Alternatively, external devices may be used to measure the vitalparameters of the user. The current emotional state of the gameplay canbe determined based on an analysis of a text content of the game. Speechand text recognition techniques may be used to perform the analysis ofthe current stress level and the current emotional state of the user.

It should be noted that the same game soundtrack may have multipledifferent classifications over time. For example, the first five secondsof the game soundtrack may be mellow, the next ten seconds may beaggressive, etc. Therefore, the classifying of the game soundtrack mayinclude classifying the portions of the game soundtrack. Similarly, thereplacement soundtrack also may be classified by classifying theportions of the replacement soundtrack.

In an example embodiment, matching the classification of the replacementsoundtrack with the classification of the game soundtrack may includematching an aggressive segment of the replacement soundtrack with anaggressive segment of the game soundtrack. Matching BPM and fading inand fading out of the soundtracks may be used to facilitate a smoothtransition of the replacement soundtrack. Additionally, multipleportions of the game soundtrack may be replaced with a single portion ofthe replacement soundtrack. Alternatively, a single portion of the gamesoundtrack may be replaced with multiple portions of the replacementsoundtrack.

In a further example embodiment, when the game loops (i.e., when theuser attempts to complete a level of the game and fails several times),the modified game music data may be different for each attempt of theuser. In other words, different replacement soundtracks may be used foreach attempt of the user to complete the level in the game. Thereplacement soundtracks in the second and further attempts of the usermay be selected based on the feedback collected during the first attemptand the current attempt.

Generally, the machine learning system (or neural network) can employone or more layers, including an input layer, an output layer, and oneor more hidden layers. At each layer (except the input layer), an inputvalue is transformed in a non-linear manner to generate a newrepresentation of the input value. The output of each hidden layer canbe used as an input to the next layer in the network, i.e., the nexthidden layer or the output layer. Each layer of the network generates anoutput from a received input in accordance with current values of arespective set of parameters. The machine learning system can be trained(learned) based on historical user data to improve accuracy of thedetermination of the classification. The historical user data caninclude data characterizing previous successful and failed attempts ofthe classification.

At operation 320, the game music data may be dynamically modified whilethe user is playing the video game. The modification may be performed byreplacing at least one of the plurality of game soundtracks with the oneor more replacement soundtracks to obtain modified game music data.

The method 300 may further include identifying sound triggers based on agameplay of the video game, actions of the user in the video game, oractions of further users in the video game. The replacement of the atleast one of the plurality of game soundtracks with the one or morereplacement soundtracks may be performed at times associated with thesound triggers. Alternatively, the replacement of the game soundtrackscan be performed upon identifying certain actions of the user in thegame or actions of other users in the game.

Additionally, the method 300 may include searching in the historicaldata database for one or more users having parameters similar toparameters of the user. Based on the search, one or more gamesoundtracks recommended or preferred by the one or more users may beselected. The dynamic modification of the game music data may furtherinclude replacing at least one of the plurality of game soundtracks withthe one or more game soundtracks recommended or preferred by the one ormore users.

At operation 325, the modified game music data may be provided to theuser while the user is playing the video game. The modified game musicdata may include the one or more game soundtracks and the one or morereplacement soundtracks. It should be noted that operations 305-335 areperformed “on the fly” such that the replacement of the game soundtrackwith the replacement soundtrack is performed dynamically and selectivelyduring the gameplay.

Optionally, an identifier of the one or more replacement soundtracksused to replace the at least one of the plurality of game soundtrack maybe temporarily displayed within the video game at operation 330. In anexample embodiment, the identifier may include the pop-up showing atleast a title of a song provided to the user in the modified game musicdata. In an example embodiment, the identifier may include severalfields, such as a name of the soundtrack currently playing, a recenthistory of soundtracks, a “thumbs up” button and a “thumbs down” buttonto enable the user to show whether the user likes or dislikes thesoundtrack, a button to enable/disable pop-ups to show the title of anew soundtrack, and the like. In some embodiments, the pop-up may beintelligently adjusted based on the current gameplay of the user.Specifically, some of the fields of the pop-up may be removed or addedduring the gameplay, or the pop-up may be displayed selectively. Forexample, if the soundtrack was already played several times during thegameplay, the pop-up may not be displayed for this soundtrack. If a newsoundtrack is playing which was not heard by the user, the pop-up may bedisplayed to the user. If the user has already liked the soundtrack, the“thumbs up” button and the “thumbs down” button may not be displayed inthe pop-up.

Upon providing the modified game music data to the user, the feedbackassociated with the gameplay of the user is continuously collected inorder to further dynamically modify the modified game music datacurrently played in the game based on the feedback.

In an example embodiment, statistical data associated with the modifiedgame music data may be collected. The user may have access to thestatistical data to see soundtracks that were played when the userplayed the game, soundtracks the user liked, soundtracks that wereplayed in multiple games of the user by virtue of the soundtracks beingtagged by the user, soundtracks that were reconfirmed by the user aspreferable, soundtracks that were double tagged (i.e., liked or dislikedseveral times) by the user, and the like.

Based on the statistical data, a recommendation service can be provided.Specifically, the soundtracks played in the games the user played may berecommended to other users based on the correlation of the taste of theother users and the taste of the user (i.e., when the other users havetaste similar to the user's taste). The similar tastes may be determinedbased on similar games played by the user and the other users, similarmusic preferred by the user and the other users, and so forth.

Moreover, when the user subsequently plays the same game, modified gamemusic data may differ from the modified game music data provided to theuser during the first time the user played the game, because differentreplacement soundtracks may be selected for modifying the game musicdata. The replacement soundtracks may be selected based on the feedbackcollected during the previous gameplay and current gameplay of the user.The feedback in the current gameplay may be different in view of adifferent emotional state of the user, different game results, differentstress level of the user, and so forth, during that current gameplay.Therefore, the user may gain different game experience when playing thesame game next time.

FIG. 4 is a high-level block diagram illustrating a computing device 400suitable for implementing the methods for dynamically modifying audioplayback of a video game as described herein. Computing device 400 mayinclude, be, or be an integral part of one or more of a variety of typesof devices, such as a game console, among others. In some embodiments,computing device 400 can be regarded as an instance of electronic device105.

As shown in FIG. 4 , a computing device 400 includes one or moreprocessors 410, memory 420, one or more mass storage devices 430, one ormore portable storage devices 440, one or more output devices 450, oneor more input devices 460, one or more network interfaces 470, one ormore optional peripheral devices 480, and a communication bus 490 foroperatively interconnecting the above-listed elements. The one or moreprocessors 410 can be configured to implement functionality and/orprocess instructions for execution within the computing device 400. Forexample, the one or more processors 410 may process instructions storedin memory 420 or instructions stored on the one or more mass storagedevices 430. Such instructions may include components of an operatingsystem or software applications.

The memory 420 may be configured to store information within thecomputing device 400 during operation. The memory 420, in some exampleembodiments, may refer to a non-transitory computer-readable storagemedium or a computer-readable storage device. In some examples, thememory 420 is a temporary memory, meaning that a primary purpose of thememory 420 may not be long-term storage. The memory 420 may also referto a volatile memory, meaning that the memory 420 does not maintainstored contents when the memory 420 is not receiving power. Examples ofvolatile memories include RAM, dynamic RAM, static RAM, and other formsof volatile memories known in the art. In some examples, the memory 420is used to store program instructions for execution by the one or moreprocessors 410. The memory 420, in one example, is used by softwareapplications or mobile applications. Generally, software or mobileapplications refer to software applications suitable for implementing atleast some operations of the methods as described herein.

The one or more mass storage devices 430 can also include one or moretransitory or non-transitory computer-readable storage media orcomputer-readable storage devices. For example, the memory 420 can storeinstructions for the one or more processors 410, user music library,game applications, game music, user settings, and so forth. In someembodiments, the one or more mass storage devices 430 may be configuredto store greater amounts of information than the memory 420. The one ormore mass storage devices 430 may also be configured for long-termstorage of information. In some examples, the one or more mass storagedevices 430 include non-volatile storage elements. Examples of suchnon-volatile storage elements include magnetic hard discs, opticaldiscs, solid-state discs, flash memories, forms of electricallyprogrammable memories (electrically programmable ROMs) or electricallyerasable and programmable memories, and other forms of non-volatilememories known in the art.

The computing device 400 may also include one or more input devices 460.The one or more input devices 460 may be configured to receive inputfrom a player through tactile, audio, video, or biometric channels.Examples of the one or more input devices 460 may include a keyboard,keypad, mouse, trackball, touchscreen, touchpad, microphone, videocamera, image sensor, fingerprint sensor, or any other device capable ofdetecting an input from the player or other source, and relaying theinput to the computing device 400 or components thereof.

The one or more output devices 450 may be configured to provide outputto the player through visual or auditory channels. The one or moreoutput devices 450 may include a video graphics adapter card, display,such as liquid crystal display monitor, light emitting diode (LED)monitor, or organic LED monitor, sound card, speaker, headphones,headset, virtual reality headset, projector, or any other device capableof generating output that may be intelligible to a player. The one ormore output devices 450 may also include a touchscreen,presence-sensitive display, or other input/output capable displays knownin the art.

The computing device 400 can also include one or more network interfaces470. The one or more network interfaces 470 can be utilized tocommunicate with external devices via one or more communicationsnetworks such as a communications network or any other wired, wireless,or optical networks. The one or more network interfaces 470 may be anetwork interface card, such as an Ethernet card, an opticaltransceiver, a radio frequency transceiver, or any other type of devicethat can send and receive information.

An operating system of the computing device 400 may control one or morefunctionalities of the computing device 400 or components thereof. Forexample, the operating system may interact with the softwareapplications or mobile applications and may facilitate one or moreinteractions between the software/mobile applications and the one ormore processors 410, memory 420, the one or more mass storage devices430, the one or more input devices 460, the one or more output devices450, and the one or more network interfaces 470. The operating systemmay interact with or be otherwise coupled to software applications orcomponents thereof. In some embodiments, software or mobile applicationsmay be included in the operating system.

Thus, methods and systems for dynamically modifying audio playback of avideo game have been described. Although embodiments have been describedwith reference to specific example embodiments, it will be evident thatvarious modifications and changes can be made to these exampleembodiments without departing from the broader spirit and scope of thedisclosed application. Accordingly, the specification and drawings areto be regarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A system for dynamically modifying audio playbackof a video game, the system comprising: a processor configured to: whilea user is playing a particular video game: ascertain game music dataassociated with the particular video game; collect feedback over aperiod of time while the user is playing the particular video game, thefeedback including at least data associated with a performance of theuser in the particular video game; based on the feedback, dynamicallymodify the game music data; and provide the modified game music data tothe user.
 2. The system of claim 1, wherein the feedback furtherincludes one or more of the following: an express action of the user,evaluation of game results, an emotional state of the user, and an inputof the user, the input including at least one of a tactile input, anaudio input, a video input, and a biometric input.
 3. The system ofclaim 2, wherein the emotional state of the user includes one or more ofthe following: a biometrical response, a facial expression, a facialmotion, and a stress level.
 4. The system of claim 2, wherein theexpress action includes one or more of the following: pressing a “like”button, pressing a “thumbs up” button, pressing a “thumbs down” button,enabling a pop-up, the pop-up showing at least a title of a songprovided to the user in the modified game music data, and disabling thepop-up.
 5. The system of claim 4, wherein the processor is furtherconfigured to: access a music source, the music source including one ormore replacement soundtracks; and wherein an analyzing unit is furtherconfigured to: determine audio parameters for each of the one or morereplacement soundtracks; classify the one or more replacementsoundtracks based on audio parameters; and index the one or morereplacement soundtracks based on the classifying to provide replacementsoundtrack indexes, wherein the replacement soundtrack indexes are usedto match a classification of the one or more replacement soundtrackswith a classification of a plurality of game soundtracks.
 6. The systemof claim 5, wherein the classification of each of the plurality of gamesoundtracks and the classification of the one or more replacementsoundtracks is determined by a machine-learning system; and wherein themachine-learning system is trained based on historical user data storedin a historical data database.
 7. The system of claim 5, wherein theaudio parameters include a beats-per-minute value, a tonality value, apitch value, a timbre value, a harmonics value, a loudness value, arhythm value, a sound envelope, a music genre, a release date, and anemotional classification.
 8. A computer-implemented method fordynamically modifying audio playback of a video game, thecomputer-implemented method comprising: providing a processor, theprocessor configured to: while a user is playing a particular videogame: ascertain game music data associated with the particular videogame; collect feedback, the feedback including at least data associatedwith a performance of the user in the particular video game; based onthe feedback, determine one or more replacement soundtracks; dynamicallymodify the game music data; and provide the modified game music data tothe user.
 9. The computer-implemented method of claim 8, furthercomprising storing the feedback in a historical data database, thehistorical data database storing a plurality of feedbacks associatedwith a plurality of users collected while each of the plurality of usersis playing the particular video game.
 10. The computer-implementedmethod of claim 8, wherein the feedback further includes one or more ofthe following: an express action of the user, evaluation of gameresults, an emotional state of the user, and an input of the user, theinput including at least one of a tactile input, an audio input, a videoinput, and a biometric input.
 11. The computer-implemented method ofclaim 10, wherein the emotional state of the user includes one or moreof the following: a biometrical response, a facial expression, a facialmotion, and a stress level.
 12. The computer-implemented method of claim11, further comprising identifying the stress level of the user based onuser vital parameters or user motions.
 13. The computer-implementedmethod of claim 10, wherein the express action includes one or more ofthe following: pressing a “like” button, pressing a “thumbs up” button,pressing a “thumbs down” button, enabling a pop-up, the pop-up showingat least a title of a song provided to the user in the modified gamemusic data, and disabling the pop-up.
 14. The computer-implementedmethod of claim 8, further comprising: searching in a historical datadatabase for one or more users having parameters similar to parametersof the user; selecting one or more game soundtracks recommended orpreferred by the one or more users; and wherein the dynamicallymodifying of the game music data further includes replacing at least oneof a plurality of game soundtracks with the one or more game soundtracksrecommended or preferred by the one or more users.
 15. Thecomputer-implemented method of claim 8, further comprising: accessing amusic source, the music source including the one or more replacementsoundtracks; determining audio parameters for each of the one or morereplacement soundtracks; classifying the one or more replacementsoundtracks based on the audio parameters; and indexing the one or morereplacement soundtracks based on the classifying to provide replacementsoundtrack indexes, wherein the replacement soundtrack indexes are usedto match a classification of the one or more replacement soundtrackswith a classification of a plurality of game soundtracks.
 16. Thecomputer-implemented method of claim 15, wherein the classification ofeach of the plurality of game soundtracks and the classification of theone or more replacement soundtracks is determined by a machine-learningsystem; and wherein the machine-learning system is trained based onhistorical user data stored in a historical data database.
 17. Thecomputer-implemented method of claim 15, wherein the audio parametersinclude a beats-per-minute value, a tonality value, a pitch value, atimbre value, a harmonics value, a loudness value, a rhythm value, asound envelope, a music genre, a release date, and an emotionalclassification.
 18. The computer-implemented method of claim 8, furthercomprising identifying sound triggers based on a gameplay of theparticular video game, actions of the user in the particular video game,or actions of further users in the particular video game, whereinreplacement of at least one of a plurality of game soundtracks with theone or more replacement soundtracks is performed at times associatedwith the sound triggers.
 19. The computer-implemented method of claim 8,further comprising temporarily displaying, within the particular videogame, an identifier of the one or more replacement soundtracks used toreplace at least one of a plurality of game soundtracks.
 20. Anon-transitory processor-readable medium having instructions storedthereon, which when executed by one or more processors, cause the one ormore processors to implement a method for dynamically modifying audioplayback of a video game, the method comprising: while a user is playinga particular video game: ascertaining game music data associated with aparticular video game; collecting feedback, the feedback including atleast data associated with a performance of the user in the particularvideo game; based on the feedback, determining one or more replacementsoundtracks; dynamically modifying the game music data; and providingthe modified game music data to the user.